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Abstract. We investigate how insights from statistical physics, namely survey 

propagation, can improve decoding of a particular class of sparse error correcting codes. 
K ' We show that a recently proposed algorithm, time averaged belief propagation, is in fact 

^ ■ intimately linked to a specific survey propagation for which Parisi's replica symmetry 

breaking parameter is set to zero, and that the latter is always superior to belief 
2 ' propagation in the high connectivity limit. We briefly look at further improvements 

available by going to the second level of replica symmetry breaking. 
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f^ ' 1. Introduction 

t:j- . Error correcting codes are an important device for communicating information through 



noisy channels. We are hving in what some social commentators have called the 



O ■ "information age", with apparently ever increasing appetite for broadband internet, 

'j^ . mobile phone data transmission, satellite television, computational power etc. Hence, 

it is no surprise that such codes have are of significant practical use. 

F^ . A family of error-correcting codes based on an identification with the ever popular 

C . Ising model from statistical mechanics were proposed by Sourlas P, and novel coding 

Q , methods were examined |21 EI- A further important advance, in terms of practical 

significance, was the move to finite rate, finite length Sourlas type codes jl] - which used 

k> '. interactions between K bits and C interactions per bit. While these codes showed good 

theoretical performance, the number of successfully transmitted bits was significantly 
higher (for certain channel noises) when K > 2. However, in this case the basin of 
attraction (BOA) for a local decoding algorithm is very small meaning that decoding 
is computationally extremely challenging for reasonable code lengths. A more directly 
useful version of the above theory involves cascading Sourlas codes [5j. The basic idea 
is to use multiple values of K in the interactions. For K = 2 (i.e. 2-body interactions) 
there is an energy landscape dominated by the state which overlaps the sent message. 
Once an algorithm has found this state with good overlap, higher body interactions can 
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be invoked which increase this overlap substantially provided that the state is within 
the BOA of the high overlap state. 

Belief propagation (BP) jH], which is closely related to the Bethe approximation 
and the cavity (replica) method in finite connectivity disordered systems jTl EI, has 
been the principle algorithm used for decoding the finite rate Sourlas codes including 
the case of the cascade version. Sometimes the quenched disorder in this problem, in the 
form of corrupted bits which cause frustration in the graph of bit interactions, brings 
about a certain disruption to the BP dynamics, which can lead to deterioration of error 
correction performance. Recently, a finite temperature survey propagation algorithm 
(SP) [01 Hm HH II2j was applied to a biased 3-body Sourlas code [13]. The latter found 
that this more advanced algorithm did lead to a shift in the critical channel noise for 
computationally feasible decoding when BP did not converge. They also noted that 
averaging the beliefs in time, or time averaged belief propagation (TABP) ^l^^ could 
improve decoding in such cases as well. However, the reason for this improvement has 
not been fully clarified yet. 

The objective of this article is to pinpoint the major contributing factors for 
such improvement in practical decoding. For this, we will investigate how the overlap 
between the original and estimated messages in the first state of the cascading codes, 
for which only K = 2 interactions are considered, are influenced by the employed 
decoding algorithms when the Almeida-Thouless (AT) stability is broken fH]- For 
simplicity of numerical experiments, we will mainly focus on the zero temperature 
cases. Although the model that we will deal with does not directly accord with 
those in the preceding work |^, the two systems have a common feature that the AT 
stability is broken and, therefore, insights gained from our simpler system is probably 
useful for understanding what occurs in the biased 3-body interaction system at finite 
temperature. By applying SP to the first stage of a cascading Sourlas code, we will 
find that we can increase the overlap with the original message below the AT-transition, 
which in turn improves the chance of being within the BOA for good decoding once we 
introduce higher body interactions. It has also recently been shown that the first step of 
replica symmetry breaking (IRSB) approach leads to higher values of the magnetisation 
for disordered ferromagnetically biased ±J 2-spin Ising systems on finite connectivity 
random graphs [E]. Thus, we strongly speculate that the higher first stage overlap 
gained by introduction of the RSB ansatz in the cascading scheme is the main factor for 
the improvement of the error correcting performance. This speculation also suggests that 
algorithms based on increasing levels of replica symmetry breaking (RSB) would improve 
matters further, albeit at high computational cost. We make an initial investigation of 
this by a brief analysis of a 2RSB survey propagation (2SP: although this could rightly 
be called a survey of surveys propagation). We will also argue that TABP corresponds 
to survey propagation of a specific type for which the Parisi parameter, which specifies 
the size of subgroups of replicas in the IRSB ansatz, is set to zero and is never bettered 
by the simple BP algorithm, as is observed in CDMA multiuser detection problems 

iiHiiini. 
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This article is organized as follows. In the next section, we introduce the model 
that we will study. In sections 3 and 4, two decoding schemes based on BP and SP 
are presented in the case of vanishing temperature. In sections 5 and 6, we will show 
numerically that the SP based algorithm with zero Parisi parameter (SPO) is, in practice, 
equivalent to TABP and provides a better performance than BP when the AT stability 
is broken. This relationship of SPO to other algorithms is provided analytically in 
the vanishing code rate limit in appendix A. The general proof of this relationship, 
unfortunately, not been managed yet. We also show numerically that 2SP can lead to 
an improvement of the decoding performance in the situation where the AT stability 
broken. This implies that a further improvement can be expected by introducing a 
higher level RSB ansatz although the computational cost for decoding will be sacrificed. 
The final section is devoted to a summary. 

2. Model definitions 

We assume that we have some message source vector ^ = {^i, . . . ,^n} G {~1) 1}^ (foi" 
simplicity of exposition we work within the Ising formalism). We encode this source 
vector via a transformation to an M-bit vector J° where J° = Hiea^i ^^"^ where 
we have M > N. The rate R of such a code is given by N/M. For each set a, 
given its size, we populate it with indices at random from {!,..., A^} with each index 
appearing exactly once in any given set a, and each index appearing Ck times in each 
K body interaction. We take the channel to be a binary symmetric channel so that 
when the encoded message vector J° is passed through the channel each bit is flipped 
independently with probability 1 — p and we denote the received message by J. The 
object of any decoding algorithm is to estimate $, with some cr using information from 
J. A natural performance measure is the overlap between our estimate and the true 
message M = j^ Yli^i^i- O'^^ '^^y ^o proceed with decoding is to look for the ground 
state of the following Hamiltonian: 

M 

H{a) = -Y,JaX{^^ (1) 

a=l Ida 

Since this Hamiltonian is invariant under the transformation ai—>-(yiC,i and Ja—>-Ja Yliea ^« 
we can gauge away the original message and are left with a multispin ferromagnetically 
biased ±J spin glass model. 

For the cascading approach we define a new auxiliary Hamiltonian 'H'{cr) = 
~J2a=i'J'aY[iea^i whcre J'^ = Ja if \a\ = 2 and J^ = otherwise. The initial step 
is to find minima of 'H'{(t) which is relatively easier than finding minima of Ti. We then 
use this minima as an initial condition for local search algorithms looking for minima 
of the full Hamiltonian (HI) . 
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3. Solution at the level of replica symmetry 

An approximate method of finding the groundstate is available via belief propagation, 
which is equivalent to the cavity method at the level of replica symmetry for locally tree 
like graphs, which the present code is in the long code length limit. The same results 
can be found in the ensemble of graphs in the thermodynamic (long code length) limit 
by use of the replica method, however, since we are interested in decoding particular 
examples the former method is preferable. The method has been discussed a great deal 
in recent literature [HI 13 El HI] so we will be relatively brief We define cavity fields hi^a 
to be the field on site i (decoded estimator ai) in the absence of link a (received bit Jo) 
and the message Ua^t to be the message from received bit J a to decoded bit ai. We are 
working with a discrete energy model and a self-consistent solution to these equations is 
given by restricting the values of {ti, h} to integers. Then, the belief propagation (BP) 
update equations describing these variables are: 

e^a= E «^-' (2) 

<^^ = ^^miJa n ^^a) (3) 

where A/'(i) denotes the set {a : i & a} and we take the convention sign(O) = 0. Other 
than the trivial paramagnetic state (/ij_»a = Ua^i = Vi, a), fixed points of the message 
passing equations Q correspond to local minima of the energy of the Hamiltonian (^. 
However, it is certainly not guaranteed that the equations will converge at all. If 
the messages converge, or after some specified number of updates if they do not, the 
estimator for the original message bits is given by 

at = sign( E Ua-,i) (4) 

a€Af(i) 

Consider a specific example, that of a code with C2 = 3 and C3 = 3 so each 
original message bit C,i is contained within three 2-body interactions and three 3-body 
interactions. Due to the 3-body interactions we find that without using the cascading 
approach we never find the state with high overlap - even at zero channel noise - due to 
the higher body interactions, which is similar to the result found for purely ferromagnetic 
3-body interactions in j2IIj, where even for these purely ferromagnetic interactions the 
structure of the interactions led to glassiness. However, if we first allow the system to 
equilibrate using the energy landscape given by only the 2-body interactions (so perform 
BP on the auxiliary Hamiltonian) we find that for a certain range of p we obtain non- 
zero values of M. Since for purely 2-body interactions there is a refiection symmetry in 
the Hamiltonian there is no bias at this stage between ±M. However, when the 3-body 
interactions are added to the Hamiltonian it is possible to infer which of these two states 
gives a better overlap overall, namely the state with lower energy. BP is then performed 
on the Hamiltonian with the 3-body interactions added. In practice, we add the the 
3-body interactions in three batches, at each batch we add one 3-body interaction to the 
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Hainiltonian for each spin (so A^/3 interactions). Since the initial state for the combined 
system is one with finite M, it is closer to the ground state with high overlap than a 
random initial condition so BP has a better chance of reaching the ground state. 

4. Solution at the level of replica symmetry breaking 

Past some critical channel noise we expect there to be an ergodicity breaking transition 
corresponding to replica symmetry breaking, at which point the update equations Q 
will no longer be sufficient to correctly characterise the system. For example, if we 
consider the initial stage of the algorithm where messages are passed purely between 
sites and 2-body links then it is known that for C2 = 3 (three 2-body interactions) 
the AT-transition occurs at p = j^ PH [TTj. Below this point a survey propagation 
algorithm [OJ Ej should improve at least the initial overlap M (the AT-point for the 
full system will not be the same as that for the initial system) - at most until the 
transition from the mixed phase (glassy ferromagnetic pure states) to the spin glass 
phase. By considering the finite temperature replica symmetric transition lines away 
from the paramagnetic phase, a bifurcation analysis of the belief propagation equations 
yields critical temperatures for the C2 = 3 regular Sourlas code: 

P^F : 1 = 2(2p - 1) tanh(/3) (5) 

P^SG: l = 2tanh2(/5) (6) 



which implies the triple point is at p = |(l + \/ 1) = 0.85355 . . .. According to the Parisi- 
Toulouse hypothesis of no re-entrance this gives us the zero temperature transition point 
between the mixed phase and the spin glass phase. 

At the first level of replica symmetry breaking, survey propagation with Parisi's 
replica symmetry breaking parameter equal to x (SPx) rather than solving for a single 
message from sites to links (and links to sites) we have to solve for a distribution of 
messages. This distribution is over the messages in different ergodic sectors. While true 
ergodicity breaking will not occur in a finite length message, the breaking effects are 
sufficiently strong to mean that it does occur to all intents and purposes on any practical 
timescale for a message of reasonable length. The key idea behind the SPx is that when 
performing an iteration of our messages, the energies in different ergodic sectors will 
vary, and as they vary the probability of finding such a given state also varies. The 
SPx equations keep track of these changes assuming that the true nature of the states is 
IRSB (which is not necessarily the case). If we again restrict ourselves to integer values 
for the messages (so in fact the integrals are really sums over the appropriate domain) 
then the iteration equations are: 



Pltl{h)^l[ n du,QU{u,)]5[h-Y,u,y\'^^-A (7) 

•^ b&N{i)\a b 

QU,{u) oc Jill dh,PJ_^^{h,)]6[u - sign(J. J] ^^•)]^''"'" (») 



jea\i 
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Figure 1. Variation (from the mean) in the overlap (solid line), energy (dashed line) 
and (zero-temperature) free energy (dotted line) versus the zero-temperature Parisi 
parameter x. The data are averages over 25 run with code lengths N = 1000. The left 
figure is for p = 0.86 while the right figure is for p — 0.88. In both figures we see that 
variation in the overlap follows variation in the energy to a reasonable extent. 



The above pair of equations are iterated until convergence (or for some sufficient number 
of iterations). At that point we can construct our estimate for the decoded message via: 



a, = sign( / dHPi{H)sign{H)) 



(9) 



MT,b^b\ 



(10) 



P,(if)oc /[ n dutQb-,^{ubmH-J2^bV 

•^ beAf(i) b 

The approach thus far has still left the choice of x. In physical problems, the general 
approach is to extremeise the generalized free energy wrt x ^0] as these states are 
the ground states of the system. However, despite using a physical approach, we are 
interested in the error correcting abilities of this system and it does not necessarily hold 
that extremising the zero temperature free energy corresponds to maximising the overlap 
M. In fact, we have found that minimizing the energy 7i(cr) gives a good guide to choice 
of X which is reasonable since our initial aim was to decode via minimizing the energy of 
the Hamiltonian (Q). If the computational effort required to fit the model is prohibitive, 
choosing a particular value of a; for SPx (we found about 0.15 was reasonable) still gives 
a marked improvement over BP in decoding across a range of channel noises p. In figure 
^we plot variation in the overlap from its mean value to that of the energy and the free 
energy (we take 10 times the variation of the free energy so that the scales are similar). 
The energy (which is observable) seems to give a reasonable indication of the overlap 
(which is not observable in practical situations). 

It is to be expected that to correctly describe state space up to the transition from a 
mixed (glassy ferromagnetic) phase to a spin glass phase full RSB needs to be taken into 
account. However, this is not feasible computationally. This does make the next obvious 
step a 2RSB algorithm, 2SPxy, where the messages will be surveys of surveys. For a 
more detailed discussion on applications to fc-SAT see j22]- We restrict our discussion 



Survey propagation for the cascading Sourlas code 7 

for simplicity to the first stage of the decoding procedure (where all interactions are 
2-body). Then we have update equations; 

nt\[P]^ I n ^QU^[Q^-H{Q}■.yr''5[P-P[{Q}■.y]\ (n) 

QU^[Q]= [dVUa[PnQ-Q[P;0]] (12) 



P[{Qhy] = rjni 1 / n d«6Q.(«6)^[/^ - J]He^^'^^"'''^^'"'t^l3) 

Q[P;a]{u)= f dhP{h)S[u~sign{Jah)] (14) 

where z[{Q};|/] is the normalising constant for P[{(5};y] and x and y are the zero 
temperature Parisi parameters. The integrals over u, h are sums over the appropriate 
integers while the integrals over measures are over distributions on the integers. We will 
compare these different approaches numerically in the following. 

5. Time averaged belief propagation and SPO 

In general there is no guarantee that BP will converge on a graph with loops. However, 
experimentally it appears that for large disordered random graphs the transition to non- 
convergence is closely linked to the AT-transition |21j. One can also rationalise this in 
terms of correlation lengths. If the correlation length is less than log(iV) then locally 
the graph appears to be a tree (apart from at a finite number of sites), so BP should 
converge provided the graph is sufficiently large (of course this is very large as the length 
scale is log(A^)). Below the AT transition due to the proliferation of pure states there 
are necessarily correlations on all length scales throughout the graph, so one would not 
expect BP to converge as it is affected by the loops. One simple, robust approach to 
deal with estimation when BP has not converged is to use time averaged BP (TABP), 
which corresponds to taking averages in time of the messages, and performing inference 
with these average messages [TU [1^1 HBj • 

Another algorithm which improves on BP but is simpler than full SPx is SPO, as 
used for the CDMA problem in ^Hl ^j- SPO does not resort to any reweighing of 
messages as the messages are passed but gains its improvement from a "majority rules" 
approach. I.e. when decoding in a region where BP does not converge, inference is 
performed at each bit by using the majority decision of the different instances. For 
the fully connected CDMA problem, the local fields are Gaussian and it is possible to 
show analytically that SPO improves inference compared to BP analytically ^^ . In the 
present situation due to the finite connectivity nature of the problem the effective fields 
are not Gaussian distributed, thus we have not been able to show that SPO improves on 
BP analytically (basically we do not have sufficiently general control of {Q{u), P{h)}). 
However, in the high C limit our present model reduces to the SK-model [2Hj and in 
this case we again find that SPO is strictly better in inference terms than BP below the 
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Figure 2. We see how the average distance D{x) between the message distributions 
from SPx and the message distributions from TABP vary with x. The solid line is for 
channel noise p = 0.88 while the dotted line is for p — 0.86. We find good support for 
the hypothesis that TABP is equivalent to SPG. 



AT-line, which is shown in appendix A. 

In fact, TABP and SPO are intuitively quite similar. The first averages over BP 
states in time, while the latter averages over different BP states in parallel from different 
initial conditions and with mixing of different solutions at each update step. Provided 
the dynamics of BP are sufficiently chaotic that it explores all states one could expect 
both averages to give similar (or even identical) results. To test this hypothesis we 
compare the distribution of local fields given by SPO to the distribution given by TABP. 
More specifically we examine the order parameter 



^(-) = ^E H DH[P^'a\\PL^a 



TABPi 



(15) 



i a£N{i) 



with the Bellinger distance Dh{P\ \Q) = 2 J dx{^/P{x) — y^Q{x))'^. In, figure|21we look 
at D{x) vs X and see that we find very good agreement for the hypothesis that SPO is 
equivalent to TABP which gives an alternative explanation to its improved performance 
over BP and provides an new method for implementing SPO. 



6. Comparison of the different decoding algorithms 

We have used four different algorithms for the initial decoding, BP, TABP (or SPO), 
SPx and 2SPxy, corresponding to different levels of replica symmetry breaking and 
model fitting. For each algorithm there is essentially only one pertinent question which 
is whether the overlap reached after initial decoding is sufficient to lead to good full 
decoding. While it is tempting to think of some critical M - in reality this is not 
sufficient to describe the state in full detail and other factors such as p and the decoding 
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Figure 3. The value of the initial overlap M versus the channel noise p for several 
different algorithms. The solid line is the theoretical replica symmetric result in the 
infinite system sized limit. The dashed line is the average result (over f runs) of BP 
(note agreement is good after p k, 0.915 the theoretical AT point). The dot-dashed 
line is TABP (averages over 10 runs) which improves significantly on BP for certain p. 
The three dashed lines are SPx (averaged over 10 runs and optimised over x) ± one 
standard deviation. The improvement of SPx on TABP is relatively modest. Finally 
the dot-dot-dashed line is a single run of 2SPxy with x — 0.15 and y = 0.1 (so not 
optimised) - it appears to improve on SPx for p — 0.86, 0.865, 0.87. Note that the 
predicted transition from a mixed phase to spin-glass phase is at p w 0.85355. 



algorithms themselves will all contribute to changes in the M required (in addition 
to finite size fiuctuations) to reach the high overlap state. However, due to issues of 
computational complexity we have tested all four algorithms on the initial decoding 
only. The 2SPxy algorithm has only been investigated very briefiy there, for a single 
value of X and y rather than optimising over them for each codeword received. We 
compare the results of these initial overlaps in figure 01 We see that each improvement 
in the algorithm does lead to an increased overlap after the initial stage, between the 
beginning of the mixed phase and the AT point. The 2SPxy does appear (as far as we 
can tell from the data) to give an improvement on the SPx algorithm for a limited range 
of channel noise. 

Next we look in figure |3 at the final overlap reached after full decoding with BP, 
SPO and SPx. We see that improving the quality of the algorithm leads to a marked 
improvement in the critical channel noise enlarging the space for which the cascading 
code approach will work. 
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Figure 4. The final overlap M versus p averaged over 10 runs of code length 
N = 10 000 for BP (dashed line), TABP (dot-dashed line) and SPx (dotted line). 
Each advance in algorithm leads to an improvement in the critical channel noise. 



7. Conclusion 

We have examined how taking replica symmetry breaking into account can improve 
decoding of cascading Sourlas codes. The basis behind our idea is that below the 
AT transition, the true overlap at the minima of the Hamiltonian (0) is above its 
replica symmetric estimate so increasing steps of replica symmetry breaking will serve 
to improve the overlap between the estimate of the message and the true message. We 
have also shown that TABP is closely related (if not identical) to SPO - this throws 
up two obvious areas for further investigation, one to find under what conditions these 
algorithms are identical and secondly to show that both of them are strict improvements 
on BP below the AT transition generally. Another area to improve is in terms of 
the 2SP, where further optimisation of the Parisi parameters should lead to further 
improvements, albeit at some considerable computational cost. The graph structure 
used here is not expected to be optimal - after the initial stage using higher body 
interactions (i.e. i^ > 3) should give improvements in the final overlap, optimising over 
possible architectures should also improve matters, but the principles developed here 
will help in any case where the channel noise is sufficiently high that decoding is taking 
place below the AT transition. 



Acknowledgments 

JPLH would like to thank Tommaso Castellani and Bastian Wemmenhove for helpful 
discussion of their work. YK acknowledges support by Grants-in-Aid Nos. 14084205 
and 17340116 from MEXT/JSPS, Japan. 



Survey propagation for the cascading Sourlas code 11 

Appendix A. Superiority of SPO to BP in dense connectivity limit 

Here, we show analytically that SPO is never bettered by BP in terms of the error 
correction performance when the number of connections tends to infinity. Such a 
property is likely to hold in the case of sparse connectivity as well, analytical proof 
of which has not yet been managed. 

Appendix A.l. SPx for densely connected systems 

For simplicity and generality, we focus on regular K = 2 systems at general finite 
temperature T = f3~^. Extension to other K values and the case of vanishing 
temperature is straightforward. SPx of this system can be expressed as 

J ..;!^«\a(2cosh(/5n,)).^ V V 

where u{hj,Ja;P) = P~^ tanh~^{tanh(pja)tanh(phj)). In order to obtain a simpler 
expression for C2 = C^oo, we introduce appropriate scalings \Ja\^^ and p = 
I + o/zp ; where Jq > and J > 0. This corresponds to situations of low code 
rate communication that copes with strong noise despite restricted signal power. Let 
us denote /(i/ijP/_^^(/ij) tanh(/3/ij) = m*^^, j dhiPl^^{hi)iai\h^{[3hi) = M*_^^ and 
characterize the system utilizing two macroscopic variables C~^ Ylih£N{i)\a ^i->b ~ 
C-'Y.a^Mii)MU - iNC)-'E.,aemMU = Q\ and C-^ E.eArw\a(^U)' - 
C"^EaeArw("^La)^ ~ {NC)~^J2i,a^ii){^Ua? = <5o bascd ou the law of large 
numbers. These, in conjunction with the central limit theorem, indicate that eq. (jA.ljl 
is reduced to 

pt+i(.. i2cosH^h))^e-^^^ 

/ dh{2 cosh{l3h))^e 5a^t^ 
where 

b6A/'(t)\a 

A*+^= E ^'(^i^.-K->.)') = '^'Wi-Qo)- (A.5) 

b&^f{t)\a 

In turn, eq. ()A.3|) provides expressions of microscopic variables 

J _ Jdh{2cosh{/3h))h ^^^ tanh{f3h) 

^i^a - : TTZIt T2 5 (A. 6) 



J dh{2cosh{ph))i^e 2a' 
, _ J dh{2cosh{(3h))T^e-^'^'t^"' tanh\(3h) 
/ dh{2 cosh(/3/i)) /?e sS?^ 



(A.7) 
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Eqs. ()A.4|) - ()A.7|) constitute the simpler expression of SPx in the dense connectivity 
hmit. Utihzing m\^^ and A*, the estimate for the decoded message at the tth update 
is provided as 

cTi = sign( I dHPi{H) tanh{pH)), (A.8) 

/ dH{2 cosh{(3H))h — 5a^ 
where 0* = J2aem) ^'^^-V 

Appendix A. 2. Macroscopic dynamics 

The performance of the SPx decoding algorithm ()A.4|) - ()A.7|) can be examined by 
following time evolution of several relevant macroscopic variables fI^ |23 1211 ■ The 
maximum entropy principle, in conjunction of the law of large numbers, makes it 
possible to regard the microscopic variable (pj^a ^s Gaussian random numbers, the 
average and variance of which are provided independently of site and check indices i, 
a as E^ and F*, respectively. We assume that the gauge transformation (yi—>-criC,i and 
Ja^Ja riiea ^i ^^ already carried out, which implies that the original message is specified 
by (1, 1, ... , 1). These indicate that Q\, Qq and the macroscopic overlap to the original 
message m* = {NC)~^ J2ia€Af(i) ^i^a ^^^ represented by 

I ^^ J Du{2 cosh{p¥{u, v)p i^nh\(3¥{u, v)) 
^ J /DM(2cosh/3$*(M,t;))t 

r ^^ f jDu{2 cosh(/3'l>*(tx, v)))^ tanh(/3$*(tx, t;)) V 
° y \ /DM(2cosh(/?$*(M,t;)))f J ' 

^t^ f j,^. I DH^ cosHP^ju, v)))^ tanh(/3$^(^, v)) 
^ J ^ /DM(2cosh(/3$*(M,t;)))t 



respectively, where Dz = ^^^— and $*(M,f) = vA*-u + \J F^v + E^ . On the other 
hand, the self-averaging property J2ZII211 and eqs. ()A.10|) . (IA.11|) and ()A.12|) imply that 
F*+^ and E^^"^ are provided by 

i,a(=J\f{i) 

^N-' J2 {Jam]^,y ^ J'QI (A.13) 

E'+' = N-' Y^ l~^Za-J^rn\ (A.14) 

4,aeA/'(i) 

respectively, where ^^^ denotes the configuration average with respect to the channel 
noise and lattice configuration. The update rule for A*+^ is already provided by eq. 
iTs]) . Using these macroscopic variables, the error probability per bit for SPx, Pb{x), 
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is calculated from 

/ Du{2 cosh(/3$*(M, f )))f tanli(^<l'*(u, v)) 



Ph(x) = / DvQ 



J Du{2cosh{P^t{u,v))) 



13 






E' 



Dv = eTicl^=j, (A. 15) 

where Q{z) = 1 and for z > and ^ < 0, respectively. It may be remarked that eqs. 
flA.5|) and ()A.10|) - ()A.14|) accord with the macroscopic description of the SPx dynamics 
for the SK model 1281. 



Appendix A. 3. Comparison between SPO and SPj3(BP) 

Now, we focus on two specific Parisi parameters x = and x = (3. A distinctive property 
of these two parameter choices is that only four out of the six variables are relevant in 
eqs. (IA.5|1 and ()A.10|1 - ()A.14|1 . which can be reduced to an identical set of four equations 

G*+i = J^Q\ (A. 16) 

E*+i = Jom*, (A. 17) 

g* = f Dztanh\p{VG^z + E')), (A.18) 

m* = f Dzta-nh{p{VG^z + E^)), (A.19) 

where G* = A* + F\ Q* = Q\ for x = while G* = F\ Q* = Q* for x = (3. As 
A* is a non-negative variable, this implies that F*, which indicates the variance of the 
local fields, for x = (3 cannot be smaller than that for a; = while E^, which represents 
the signal strength of the original message in the local fields, is identical in both cases. 
Applying this result to eq. ()A.15|) leads to a conclusion that SPO does not provide worse 
performance than SP/3. 

In terms of the decoding problem, SP/3 and BP provide identical performance. 
Therefore, the above argument indicates that SPO is never overcome by BP with respect 
to the error correction performance. In order to show this, we characterise the auxiliary 
distributions P/_^a(/i) and Q\^i{u) utilising the first moments of tanh(/3/i) and tanh(/5M), 
respectively, as 

HUa = \ tanh"^ ( j dhPl^^Qi) tanh(/3/i) J , (A.20) 

UU = ^ tanh-i (^j dhQU^iu) tanh(/3«)^ . (A.21) 

Although higher moments are necessary for completely describing general SPx dynamics, 
eqs. (IA.1|) and ()A.2|) indicate that for the specific choice oi x = P, Hj^^ and f/^^j 
constitute a closed update rule as 

Hltl= E ^^-' (A.22) 

UU,=u{H]^,,Ja^(3), (A.23) 
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which is nothing but the BP dynamics of finite temperature T = (3~^ and are reduced 
to eqs. (j21) and (jS)) in the zero temperature hmit (3—^oo. This imphes that as long as 
only the first moment is concerned, which is the case for the current decoding problem, 
SP/3 and BP are equivalent. 
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